Image evaluation apparatus, image evaluation method, program, and integrated circuit

ABSTRACT

In the image evaluation apparatus, a clothing recognition unit  106  performs, for each person appearing in each of images included in an image group generated by an image group generation unit  102,  recognition of clothing that the person is wearing. An image event evaluation unit  107,  according to types of clothing recognized by the clothing recognition unit  106  and a frequency of appearance of each type of clothing in the images in the image group, collectively evaluates the images included in the image group.

TECHNICAL FIELD

The present invention relates to an image evaluation apparatus thatevaluates images by making use of information pertaining to clothing.

BACKGROUND ART

Digital image photography devices such as digital still cameras andmobile phones having camera functions have gained much popularity. Asrecording media for storing images taken by using such digital imagephotography devices, high-capacity recording media, such as hard disks,are being provided to users.

A user, when using a high-capacity recording medium, is able to store alarge number of images. However, when a user stores a large number ofimages to a high-capacity recording medium, the user experiencesdifficulty in searching for a desired image among the large number ofimages stored to the high-capacity recording medium.

As a method for enabling a user to easily find a desired image fromamong a large number of images, a method is known of organizing imagesby performing classification of the images according to the events atwhich the images were taken (here, the term “events” refer to suchevents as a school entrance ceremony and a school sports day). As oneexample of such a method, Patent Literature 1 introduces a method ofperforming recognition of the clothing that people appearing in theimages are wearing, performing an evaluation of determining the eventsat which the images were taken according to the results of therecognition, and classifying the images according to the results of theevaluation.

When performing recognition of the clothing that a person appearing inan image is wearing, it is necessary to detect a region of the imagedeterminable as corresponding to the clothing that the person is wearing(such a region is hereinafter referred to as a “clothing region”) and toextract image characteristics from the clothing region so detected.Examples of image characteristics extractable from a clothing regioninclude a ratio of colors in the clothing region and a value indicativeof change in luminance between adjacent pixels in the clothing region.

CITATION LIST Patent Literature [Patent Literature 1]

Japanese Patent Application Publication No. 2009-301119

SUMMARY OF INVENTION Technical Problem

However, when performing recognition by using image characteristics, itis difficult to accurately detect a clothing region in an image sincethe clothing region changes when the pose of the corresponding personchanges. In addition, the image characteristics extractable from aclothing region change according to the direction that the correspondingperson is facing and/or the illumination used. As such, there are caseswhere recognition of the clothing that a person appearing in an image iswearing cannot be performed correctly when performing recognition byusing image characteristics.

Due to this, an apparatus that evaluates an image according to theclothing that a person appearing in the image is wearing may evaluatethe image incorrectly when recognition of the clothing that the personappearing in the image is wearing is performed incorrectly.

In view of such a problem, the present invention provides an imageevaluation apparatus that is capable of evaluating an image correctlyeven when recognition of the clothing that a person appearing in theimage is wearing is performed incorrectly.

Solution to the Problems

One aspect of the present invention is an image evaluation apparatusthat evaluates images in an image group, comprising: a firstspecification unit that specifies, for each person appearing in each ofthe images, a type of clothing that the person is wearing as a firstclothing type; a second specification unit that specifies a type ofclothing characterizing the image group as a second clothing typeaccording to a frequency of appearance of each of different firstclothing types specified from the images in the image group; and anevaluation unit that evaluates the images in the image group bydetermining, according to the second clothing type, an event at whichthe images in the image group have been taken.

Advantageous Effects of the Invention

According to the image evaluation apparatus pertaining to one aspect ofthe present invention, even when recognition of the clothing that aperson appearing in a given image is wearing is performed incorrectly,evaluation of the given image is performed correctly due to referencebeing made to the frequency at which different types of clothing appearin a plurality of images including the given image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an image evaluation apparatus100 in embodiment 1.

FIG. 2 illustrates one example of an image management information table.

FIG. 3 illustrates one example of a people management information table.

FIGS. 4A though 4C illustrate three example images, face regionsdetected in the example images, and clothing regions detected in theexample images.

FIGS. 5A and 5B illustrate two example images, face regions detected inthe example images, and clothing regions detected in the example images.

FIG. 6 is a flowchart illustrating processing performed by the imageevaluation apparatus 100 in embodiment 1.

FIG. 7 is a flowchart illustrating processing performed by an imageevent evaluation unit 107 in embodiment 1.

FIG. 8 illustrates one example of a table indicating third clothingtypes and the number of images characterized by each third clothingtype.

FIG. 9 illustrates one example of a table indicating a correspondencebetween clothing types and events.

FIG. 10 illustrates one example of a table indicating results of eventevaluation performed with respect to image groups.

FIG. 11 is a functional block diagram of an image evaluation apparatus1100 in embodiment 2.

FIG. 12 illustrates one example of a table indicating similarity betweenclothing that people appearing in an image are wearing.

FIG. 13 is a flowchart illustrating processing performed by the imageevaluation apparatus 1100 in embodiment 2.

FIG. 14 is a flowchart illustrating processing performed by an imageevent evaluation unit 1102 in embodiment 2.

FIG. 15 is a functional block diagram of an image evaluation apparatus1500 in embodiment 3.

FIG. 16 illustrates one example of a table indicating faces and acluster corresponding to each face.

FIG. 17 is a flowchart illustrating processing performed by the imageevaluation apparatus 1500 in embodiment 3.

FIG. 18 is a flowchart illustrating processing performed by an imageevent evaluation unit 1503 in embodiment 3.

FIG. 19 illustrates one example of a table indicating clusters and facesbelonging to each cluster.

FIG. 20 illustrates one example of a table indicating faces belonging toa cluster and a first clothing type for each face.

FIG. 21 is a functional block diagram of an image evaluation apparatus2100 in embodiment 4.

FIG. 22 is a flowchart illustrating processing performed by the imageevaluation apparatus 2100 in embodiment 4.

FIG. 23 is a flowchart illustrating processing performed by an imageevent evaluation unit 2101 in embodiment 4.

FIG. 24 illustrates one example of a table indicating third clothingtypes and the number of clusters characterized by each third clothingtype.

DESCRIPTION OF EMBODIMENTS <Overview of Invention>

A person taking images usually takes a plurality of images at one event.When conventional technology is applied, an image taken at a given eventcan be correctly evaluated as having been taken at the given event whenthe clothing appearing in the image is recognized correctly. On theother hand, it is difficult to correctly evaluate an image taken at agiven event as having been taken at the given event when the clothingappearing in the image is not recognized correctly. According to thepresent invention, an image can be evaluated correctly even whenrecognition of the clothing appearing in the image is performedincorrectly. This owes to information pertaining to another image whoseclothing has been correctly recognized being used in the evaluation ofthe image.

Embodiment 1

In the following, description is provided on an image evaluationapparatus 100 pertaining to one embodiment of the present invention,with reference to the accompanying figures.

<Structure>

FIG. 1 is a functional block diagram of the image evaluation apparatus100 in embodiment 1. As illustrated in FIG. 1, the image evaluationapparatus 100 is connected with a photography device 120 and a displaydevice 130.

The image evaluation apparatus 100 acquires a group of images from thephotography device 120, performs evaluation of the images, and outputsthe images in accordance with the results of the evaluation to thedisplay device 130.

The photography device 120 photographs images and accumulates theimages. The photography device 120 is, for instance, implemented as adigital camera or the like, and is connected with the image evaluationapparatus 100 via a Universal Serial Bus (USB) cable or the like.

The display device 130 displays images such as those output from theimage evaluation apparatus 100. The display device 130 is, for instance,implemented as a digital television or the like, and is connected withthe image evaluation apparatus 100 via a High Definition MultimediaInterface (HDMI) cable or the like.

In the following, description is provided on a functional structure ofthe image evaluation apparatus 100 pertaining to the present embodiment.The image evaluation apparatus 100 includes an image informationacquisition unit 110, an image event evaluation unit 107, and a storageunit 108. The image information acquisition unit 110 includes an imageacquisition unit 101, an image group generation unit 102, a facedetection unit 103, a clothing detection unit 104, a clothingcharacteristics extraction unit 105, and a clothing recognition unit106.

The image acquisition unit 101 collectively acquires a group of imagesaccumulated by the photography device 120, and provides each of theacquired images with a unique image ID. The image acquisition unit 101registers the image IDs provided to the acquired images to an imagemanagement information table 201 illustrated in FIG. 2. The imagemanagement information table 201 is stored in the storage unit 108.

The image group generation unit 102 classifies the images acquired bythe image acquisition unit 101 into a plurality of image groups. Theimage group generation unit 102 generates image groups by, for instance,classifying images having been taken on the same date into the sameimage group. The image group generation unit 102 provides each imageclassified into a given image group with an image group ID uniquelycorresponding to the given image group. Specifically, the image groupgeneration unit 102 acquires a photography date/time of each of theacquired images from Exchangeable Image File Format (EXIF) informationprovided to each of the acquired images and uses the EXIF information soas to classify the acquired images into image groups. The image groupgeneration unit 102 registers, for each of the acquired images, aphotography date/time and an image group ID of the image group intowhich the image has been classified to the image management informationtable 201.

The face detection unit 103 detects, in each of the images acquired bythe image acquisition unit 101, a square region (represented bycoordinates in the image) corresponding to each person's face appearingin the image (hereinafter referred to as a “face region”). The facedetection unit 103 provides each of the faces detected from the acquiredimages with a unique face ID. The face detection unit 103 registers theface IDs to the image management information table 201 and a peoplemanagement information table 301 illustrated in FIG. 3. The peoplemanagement information table 301 is stored in the storage unit 108. Thedetection of face regions in the acquired images is performed throughmatching by using a face learning dictionary. The face learningdictionary is prepared in advance and is composed of a plurality of faceimages.

The clothing detection unit 104 detects, for a face region having beendetected in a given one of the acquired images by the face detectionunit 103, a region in the given image in which the clothing worn by theperson corresponding to the face region appears (hereinafter referred toas a “clothing region”). The detection of a clothing region is performedaccording to coordinates of the corresponding face region. Specifically,a clothing region can be calculated from (i) a position and a size of acorresponding face region and (ii) a predetermined ratio between theface, the neck, and the upper half of the body of people appearing inthe images. The clothing detection unit 104 manages each clothing regionso detected in association with the corresponding face ID.

In the following, description is provided on a specific example of amethod of calculating a clothing region by using a face region, withreference to FIGS. 4A, 4B, and 4C. As in image 401 illustrated in FIG.4A, when a face region detected by the face detection unit 103 has asize of 1.0×1.0 (height×width), a corresponding clothing region isdefined as a region having a size of 2.8×2.0 (height×width) located 0.2lower in a height direction of the image from the lower end of the faceregion. That is, for instance, when a face region detected by the facedetection unit 103 has a size of 100 pixels×100 pixels (height×width), acorresponding clothing region is a region having a size of 280×200pixels (height×width) located 20 pixels lower in the height directionfrom the lower end of the face region.

Since a clothing region is calculated automatically according to acorresponding face region, there are cases as illustrated in image 402in FIG. 4B where a region detected as a clothing region extends overimage boundaries. In such a case, a portion of the detected clothingregion lying within the image boundaries is determined as a clothingregion. For instance, in the example illustrated in FIG. 4B, only theshaded portion of the detected clothing region 402 a′, which lies withinthe image boundaries, is actually determined as a clothing region.

Further, when a region detected as a clothing region for a given personoverlaps a face region of another person as in the case of image 403illustrated in FIG. 4C, a portion of the detected clothing region thatdoes not overlap the face region of the other person is actuallydetermined as the clothing region for the given person. In addition,when two or more regions detected as clothing regions overlap each otheras also illustrated in image 403 in FIG. 4C, assuming that a firstperson whose face region appears lower in the height direction of animage appears in front, in the camera direction, of a second personwhose face region appears higher in the height direction of the image,the clothing region for the second person is determined as a portion ofthe detected clothing region not overlapping the clothing region for thefirst person. This is based on the assumption that, when the firstperson appearing in front of the second person in the image is tallerthan the second person, the second person would be hidden by the firstperson in the image, and hence, the face region of the second personwould not be detected in the image.

In the example illustrated in image 403 illustrated in FIG. 4C, two faceregions, namely face region 403 a and face region 403 b, are detected.When comparing the two face regions so detected, the face region 403 bis detected lower in the height direction of the image 403 than the faceregion 403 a. As such, when clothing regions 403 a′ and 403 b′respectively corresponding to the face regions 403 a and 403 b overlapeach other, the clothing region 403 b′ corresponding to the face region403 b is preferentially detected. With regards to the clothing regioncorresponding to the face region 403 a, a cross-hatched portion of theclothing region 403 a′, which does not overlap the clothing region 403b′, is determined as the clothing region corresponding to the faceregion 403 a.

FIGS. 5A and 5B illustrate two example images, face regions detected inthe example images by the face detection unit 103, and clothing regionsdetected in the example images by the clothing detection unit 104. InFIGS. 5A and 5B, regions enclosed by solid lines indicate face regions,and regions enclosed by dotted lines indicate clothing regionscorresponding to the face regions having been detected.

Returning to FIG. 1, the clothing characteristics extraction unit 105extracts image characteristics for the clothing regions detected by theclothing detection unit 104. In general, image characteristics indicatecharacteristics of a distribution of pixel values pertaining to aplurality of pixels in an image. Here, image characteristics for aclothing region include, for instance, a ratio of colors appearingwithin the clothing region and an amount of change in luminance betweenadjacent pixels in the clothing region. The clothing characteristicsextraction unit 105 registers the image characteristics extracted fromthe clothing regions to the people management information table 301.

The clothing recognition unit 106, according to the imagecharacteristics for each clothing region registered to the peoplemanagement information table 301, determines the type of clothing(hereinafter simply referred to as a “clothing type”) appearing in theclothing region. The determination of clothing types is performed byproviding a classifier that is capable of determining clothing typesaccording to image characteristics and causing the classifier todetermine a clothing type corresponding to each face ID. The classifieris provided with the capability of determining clothing types by beingsubjected to learning according to the Support Vector Machine (SVM)method in advance. The clothing types of the clothing regionscorresponding to the face IDs, recognition of which is performed by theclothing recognition unit 106, are referred to as first clothing types.The clothing recognition unit 106 registers the first clothing typeshaving been specified to the people management information table 301.For instance, in FIG. 3, the first clothing types for the peopleidentified by face IDs 2 and 3 commonly indicate a “school gym uniform”clothing type. It should be noted that there may be cases where theclothing that a person identified by a given face ID is wearing isspecified as not belonging to any clothing type, as is the case with theperson identified by face ID 1 in FIG. 3.

The image event evaluation unit 107 performs event evaluation withrespect to each image group according to the contents of the imagemanagement information table 201 and the contents of the peoplemanagement information table 301. Details of how event evaluation isperformed are described later. Note that here, “event evaluation” refersto an evaluation of associating an evaluation-target image group or anevaluation-target image with a corresponding event.

The storage unit 108 stores: the image management information table 201indicating, for each image ID, a corresponding photography date/time, acorresponding image group ID, and a face ID of each face appearing inthe image; the people management information table 301 indicating, foreach face ID, image characteristics extracted from a correspondingclothing region and a corresponding first clothing type; andlater-described tables 801, 901, and 1001 respectively illustrated inFIGS. 8, 9, and 10. The table 801 indicates third clothing types and thenumber of images characterized by each third clothing type, the table901 indicates a correspondence between clothing types and events, andthe table 1001 indicates results of event evaluation performed withrespect to image groups.

The image evaluation apparatus 100 includes a processor and a memorythat are not illustrated in FIG. 1, and each of the functional units ofthe image evaluation apparatus 100 are realized by the processorexecuting a program stored to the memory.

<Operations>

In the following, description is provided on operations of the imageevaluation apparatus 100 pertaining to the present embodiment, withreference to the flowchart illustrated in FIG. 6.

First, the image acquisition unit 101 acquires images accumulated by thephotography device 120, and registers an image ID uniquely provided toeach of the acquired images to the image management information table201 (Step S601).

The image group generation unit 102 generates image groups from theimages acquired by the image acquisition unit 101, and registers animage group ID uniquely provided to each image group to the imagemanagement information table 201 (Step S602).

The face detection unit 103 detects, in each of the acquired images,face regions each corresponding to a person's face, and registers a faceID uniquely provided to each of the faces to the image managementinformation table 201 and the people management information table 301(Step S603).

The clothing detection unit 104 detects, according to the face regionsdetected by the face detection unit 103, a clothing region correspondingto each face region (Step S604).

The clothing characteristics extraction unit 105 extracts imagecharacteristics for each clothing region detected by the clothingdetection unit 104, and registers image characteristics corresponding toeach of the clothing regions to the people management information table301 (Step S605).

The clothing recognition unit 106 specifies a first clothing type foreach person appearing in each of the acquired images according to theimage characteristics for the clothing regions registered to the peoplemanagement information table 301, and registers the results of thespecification to the people management information table 301 (StepS606).

The image event evaluation unit 107 performs event evaluation of each ofthe image groups based on the contents of the image managementinformation table 201 and the contents of the people managementinformation table 301, which are created through the execution of theprocessing in Steps S601 through S606 (Step S607). FIG. 7 is a flowchartillustrating details of Step S607, where the image event evaluation unit107 performs event evaluation with respect to one evaluation-targetimage group.

The image event evaluation unit 107 selects, from among the images inthe evaluation-target image group, images each including at least apredetermined number of people (Step S701). Here, the image eventevaluation unit 107 selects, from among the images in theevaluation-target image group, images each including at least two peopleso as to perform event evaluation with respect to the images in theevaluation-target image group according to the clothing that a pluralityof people appearing in the images are wearing. The number of peopleappearing in each image is determinable from the number of face IDsregistered to a face ID field of the image management information table201.

The image event evaluation unit 107 selects one image from among theimages selected in Step S701, and specifies, as a clothing typecharacterizing the image (hereinafter referred to as a “third clothingtype”), a first clothing type satisfying a first criterion in the image(Step S702). Here, a given first clothing type is determined assatisfying the first criterion when a ratio of the number of clothing inthe image corresponding to the given first clothing type to the totalnumber of clothing appearing in the image exceeds 0.5. That is, whenmore than half of the people appearing in the image are commonly wearingclothing of a given clothing type, the image event evaluation unit 107specifies the given clothing type as the third clothing typecharacterizing the image.

In the following, explanation is provided of a specific example of theoperations involved in Step S702, with reference to the image managementinformation table 201 illustrated in FIG. 2 and the people managementinformation table 301 illustrated in FIG. 3. In this example, an imageidentified by image ID 1 is selected in Step S702 by the image eventevaluation unit 107. Further, according to the image managementinformation table 201, four people identified by face IDs 1 through 4appear in the image. In addition, according to the people managementinformation table 301, the first clothing types for the three peopleidentified by face IDs 2 through 4 commonly indicate the “school gymuniform” clothing type while the first clothing type for the personidentified by face ID 1 indicates an “unspecified” clothing type. Here,when a first clothing type for a given person indicates the“unspecified” clothing type, the clothing type for the given person is aclothing type specified as not belonging to any specific clothing type.As such, the ratio of the number of people in the image wearing schoolgym uniforms to the total number of people appearing in the image iscalculated as 3/4=0.75. Since the ratio (0.75) of the number of peoplewearing school gym uniforms (3) to the total number of people appearingin the image (4) exceeds 0.5, the “school gym uniform” clothing typesatisfies the first criterion. As such, the image event evaluation unit107 specifies the “school gym uniform” clothing type as the thirdclothing type for the image identified by image ID 1.

Subsequently, the image event evaluation unit 107 counts the number ofimages characterized by the same third clothing type (Step S703).Specifically, the image event evaluation unit 107 creates the table 801illustrated in FIG. 8 indicating third clothing types and the number ofimages characterized by each third clothing type. Further, when a giventhird clothing type is specified as characterizing a given image in StepS702, the image event evaluation unit 107 increments, by one, the valuein the field indicating the number of images characterized by the givenclothing type.

Following this, the image event evaluation unit 107 determines whetheror not the above-described processing has been performed with respect toall of the image having been selected in Step S701 (Step S704).Processing proceeds to Step S705 when determining affirmatively in StepS704 while processing returns to Step S702 when determining negativelyin Step S704.

When determining affirmatively in Step S704, the image event evaluationunit 107 specifies a clothing type characterizing the image group(hereinafter referred to as a “second clothing type”) by specifying aclothing type satisfying a second criterion in the image group (StepS705). Here, a given third clothing type is determined as satisfying thesecond criterion when a ratio of the number of images characterized bythe given third clothing type to the total number of images in the imagegroup selected in Step S701 exceeds 0.5. That is, when more than half ofthe people appearing in an image are wearing clothing of a givenclothing type in more than half of the images in the image group inwhich at least two people appear, the image event evaluation unit 107specifies the given clothing type as the second clothing typecharacterizing the image group.

In the following, description is provided on a specific example of theprocessing in Step S705, with reference to the table 801 illustrated inFIG. 8 indicating third clothing types and the number of imagescharacterized by each third clothing type. In this example, fifty imageshave been selected in Step S701 from among the images in the imagegroup. Further, the ratio of the number of images characterized by the“school gym uniform” clothing type to the total number of imagesselected in S701 is calculated as 36/50=0.72. Since the ratio (0.72) ofthe number of images characterized by the “school gym uniform” clothingtype (36) to the total number of images selected in S701 (50) exceeds apredetermined ratio (0.5), the “school gym uniform” clothing typesatisfies the second criterion. As such, the image event evaluation unit107 specifies the “school gym uniform” clothing type as the secondclothing type for the evaluation-target image group.

The image event evaluation unit 107 performs event evaluation withrespect to the image group according to the second clothing type (StepS706). Specifically, the image event evaluation unit 107 specifies anevent associated with the second clothing type for the image group byreferring to the table 901 stored in advance to the storage unit 108,and associates the image group with the specified event. The table 901,which is illustrated in FIG. 9, indicates the correspondence betweenclothing types and events. In addition, the image event evaluation unit107 provides each of the images belonging to the image group with anevent tag indicating the specified event.

The image evaluation apparatus 100 performs event evaluation asdescribed above with respect to all of the image groups having beengenerated. FIG. 10 illustrates an example of results of event evaluationperformed with respect to image groups.

Finally, the image evaluation apparatus 100 outputs each of the imagesacquired by the image acquisition unit 101 to the display device 130such that the results of the evaluation performed with respect to theimages are displayed by the display device 130. For instance, the imageevaluation apparatus 100 outputs each image to the display device 130such that an event name of an event tag provided to the image iscomposed with the image.

<Conclusion>

The image evaluation apparatus 100 pertaining to the present embodimentperforms event evaluation of images included in an image groupconstituted of at least two images according to the frequency ofappearance, in the image group, of images characterized by eachdifferent clothing type.

Even when recognition of clothing that a plurality of people are wearingin a small number of images in the image group is performed incorrectly,the image evaluation apparatus 100 pertaining to the present inventionis able to correctly evaluate the small number of images with respect towhich recognition of clothing has been performed incorrectly providedthat the recognition of clothing that people are wearing is correctlyperformed for a majority of the images in the image group. In short, theimage evaluation apparatus 100 pertaining to the present embodiment iscapable of evaluating images with higher accuracy compared to whenevaluation is performed of images one-by-one.

Embodiment 2

In embodiment 1, event evaluation of images is performed based on onlythe information pertaining to the clothing appearing in the images,recognition of which is performed by the clothing recognition unit 106.In embodiment 2, event evaluation of images is performed by using, inaddition to the information utilized in embodiment 1, the similaritybetween clothing worn by people appearing in the images based on theassumption that, when the clothing that one person in an image iswearing is similar to the clothing that another person in the image iswearing, the clothing worn by the two people belong to the same clothingtype. Note that in the following, structures and data similar to thosein embodiment 1 are provided with the same reference signs anddescription thereon is omitted.

<Structure>

In the following, description is provided on an image evaluationapparatus 1100 pertaining to the present embodiment. FIG. 11 is afunctional block diagram of the image evaluation apparatus 1100 inembodiment 2. In the image evaluation apparatus 1100, the image eventevaluation unit 107, the storage unit 108, and the image informationacquisition unit 110 in embodiment 1 are respectively replaced with animage event evaluation unit 1102, a storage unit 1103, and an imageinformation acquisition unit 1110. The image information acquisitionunit 1110 includes, in addition to the structure of the imageinformation acquisition unit 110, a similarity degree calculation unit1101.

The similarity degree calculation unit 1101 calculates, for eachcombination of two people appearing in a given image, a similaritydegree indicating similarity between a combination of clothing worn bythe two people according to the image characteristics pertaining toclothing registered to the people management information table 301. Thesimilarity degree calculation unit 1101 registers the similarity degreefor each combination of clothing in a given image to a table 1201illustrated in FIG. 12, which indicates similarity between clothing thatpeople in an image are wearing. The table 1201 is held by the storageunit 1103. Specifically, a similarity degree between a combination ofclothing appearing in an image is calculated as a cosine similaritybetween two vectors each representing image characteristics of clothingworn by a corresponding person.

The image event evaluation unit 1102 performs event evaluation of eachimage group based on the contents of the image management informationtable 201, the contents of the people management information table 301,and the contents of the table 1201 indicating similarity between theclothing that people in an image are wearing. Details of how eventevaluation is performed are described later.

The storage unit 1103 stores the table 1201 indicating similaritybetween the clothing that people in an image are wearing, in addition tothe image management information table 201, the people managementinformation table 301, the table 801 indicating third clothing types andthe number of images characterized by each third clothing type, thetable 901 indicating the correspondence between clothing types andevents, and the table 1001 indicating results of event evaluationperformed with respect to image groups.

<Operations>

In the following, description is provided on operations of the imageevaluation apparatus 1100 pertaining to the present embodiment, withreference to the flowchart illustrated in FIG. 13. In the following,description on the processing performed in each of Steps S601 throughS606 is omitted for being similar as the corresponding processing inembodiment 1.

The similarity degree calculation unit 1101 calculates a similaritydegree for each combination of clothing appearing in each imageaccording to the image characteristics pertaining to clothing registeredto the people management information table 301 (Step S1301).

The image event evaluation unit 1102 performs event evaluation of imagegroups based on the contents of the image management information table201, the contents of the people management information table 301, andthe contents of the table 1201 indicating similarity between theclothing that people in an image are wearing (Step S1302). FIG. 14 is adetailed flowchart illustrating operations involved in Step S1301, wherethe image event evaluation unit 1102 performs event evaluation of oneevaluation-target image group.

The image event evaluation unit 1102 selects, from among images includedin the evaluation-target image group, images each including at least apredetermined number of people (Step S701).

The image event evaluation unit 1102 selects one image from among theimages selected in Step S701, and determines whether or not combinationsof clothing indicating similarity, among the clothing appearing in theselected image, satisfy a third criterion (Step S1401). Processingproceeds to Step S1403 when determining affirmatively in Step S1401while processing proceeds to Step S1402 when determining negatively inStep S1401. Here, the expression “combination of clothing indicatingsimilarity” refers to a combination of clothing worn by two peopleappearing in the image whose similarity degree, calculated by thesimilarity degree calculation unit 1101, exceeds 0.7. Here, combinationsof clothing indicating similarity in an image satisfy the thirdcriterion when a ratio of the number of the combinations of clothingindicating similarity to the total number of possible combinations ofclothing appearing in the image exceeds 0.6.

In the following, description is provided on a specific example of theprocessing in Step S1401, with reference to the table 1201 indicatingsimilarity between clothing that people in an image are wearingillustrated in FIG. 12. More specifically, the table 1201 in FIG. 12indicates similarity degrees for combinations of clothing worn by peopleappearing in an image identified by image ID 1. Four people identifiedby face IDs 1 through 4 appear in the image. As such, the total numberof possible combinations of clothing in the image is calculated as₄C₂=(4×3)/(2×1)=6. In addition, according to the table 1201 indicatingsimilarity between the clothing that people in an image are wearing, thenumber of combinations of clothing indicating similarity in the image isthree, or that is, three combinations of face IDs, namely thecombinations (2, 3), (2, 4), and (3, 4), correspond to combinations ofclothing having similarity degrees exceeding the predetermined value0.7. As such, the ratio of the number of combinations of clothingindicating similarity to the total number of possible combinations ofclothing in the image is calculated as 3/6=0.5. Since the ratio (0.5) ofthe number of combinations of clothing indicating similarity (3) to thetotal number of possible combinations of clothing in the image (6) issmaller than a predetermined value (0.6), the combinations of clothingindicating similarity in the image do not satisfy the third criterion.

Subsequently, the image event evaluation unit 1102 determines whether ornot the combinations of clothing indicating similarity in the imageselected in Step S1401 satisfy a fourth criterion (Step S1402).Processing proceeds to Step S1403 when determining affirmatively in StepS1402 while processing returns to Step S1405 when determining negativelyin Step S1402. Here, the combinations of clothing indicating similarityin the image satisfy the fourth criterion when an average value of thesimilarity degrees for the combinations of clothing indicatingsimilarity exceeds a value calculated by using a predetermined formula0.9−(0.01×N), where N indicates the number of combinations of clothingindicating similarity.

In the following, description is provided on a specific example of theprocessing in Step S1402, with reference to the table 1201 indicatingsimilarity between clothing that people in an image are wearingillustrated in FIG. 12. Since there are three combinations of clothingindicating similarity in the image, which are identified by combinationsof face IDs (2, 3), (2, 4), and (3, 4), the combinations of clothingindicating similarity in the image satisfy the fourth criterion when theaverage value of the similarity degrees for the three combinations ofclothing exceed a value calculated as 0.9−(0.01×3)=0.87 by using thepredetermined formula. In this example, the average value of thesimilarity degrees for the three combinations of clothing indicatingsimilarity is calculated as (0.93+0.98+0.91)/3=0.94. Since the averagevalue of the similarity degrees for the combinations of clothingindicating similarity (0.94) exceeds a value (0.87) calculated by usingthe predetermined formula, the image event evaluation unit 1102determines that the combinations clothing indicating similarity in theimage satisfy the fourth criterion.

Subsequently, the image event evaluation unit 1102 specifies a clothingtype characterizing the image selected in the processing in Step S1401(i.e., a third clothing type for the image selected in the processing inStep S1401) by specifying a clothing type in the image satisfying afifth criterion (Step S1403). Here, a first clothing type appearing inthe image satisfying the fifth criterion is, for instance, a firstclothing type having been specified as belonging to a specific clothingtype by the clothing recognition unit 106 among the first clothing typesfor the clothing included in the combinations of clothing indicatingsimilarity. Here, when two or more first clothing types for the clothingincluded in the combinations of clothing indicating similaritysimultaneously satisfy the fifth criterion, the image event evaluationunit 1102 specifies, as the third clothing type for the image, a firstclothing type common to a relatively great number of the clothingincluded in the combinations of clothing indicating similarity in theimage selected through the processing in Step S1401.

In the following, description is provided on a specific example of theprocessing in Step S1403, with reference to the table 1201 indicatingsimilarity between clothing that people in an image are wearingillustrated in FIG. 12 and the people management information table 301illustrated in FIG. 3. In this example, there are three combinations ofclothing indicating similarity, which are identified by combinations offace IDs (2, 3), (2, 4), and (3, 4). Hence, the people wearing clothingincluded in the three combinations of clothing indicating similarity areidentified as the three people identified by face IDs 2, 3, and 4.According to the people management information table 301, the clothingthat each of the three people indicated by face IDs 2, 3, and 4 iswearing in the image is specified as belonging to the “school gymuniform” clothing type by the clothing recognition unit 106. As such,the image event evaluation unit 1102 specifies the “school gym uniform”clothing type as the third clothing type characterizing the image.

Following this point, the image event evaluation unit 1102 performsprocessing corresponding to the processing performed by the image eventevaluation unit 107 in Steps S703 through S706 in embodiment 1.Therefore, description concerning the processing performed by the imageevent evaluation unit 1102 following this point is provided in asimplified manner.

The image event evaluation unit 1102 counts the number of images in theimage group characterized by the same third clothing type (Step S1404).

The image event evaluation unit 1102 determines whether or not theabove-described processing has been performed with respect to all of theimages having been selected in Step S701 (Step S1405). Processingproceeds to Step S1406 when determining affirmatively in Step S1405while processing returns to Step S1401 when determining negatively inStep S1405.

When determining affirmatively in Step S1405, the image event evaluationunit 1102 specifies a clothing type characterizing the image group(i.e., the second clothing type) by specifying a clothing typesatisfying a sixth criterion in the image group (Step S1406). Here, agiven third clothing type is determined as satisfying the sixthcriterion when, for instance, a ratio of the number of imagescharacterized by the given third clothing type to the total number ofimages of the image group selected in Step S701 exceeds 0.5.

The image event evaluation unit 1102 performs event evaluation withrespect to the image group according to the second clothing type (StepS1407).

The image evaluation apparatus 1100 performs event evaluation asdescribed above with respect to all of the image groups having beengenerated. Finally, the image evaluation apparatus 1100 outputs each ofthe images acquired by the image acquisition unit 101 to the displaydevice 130 such that the results of the evaluation performed withrespect to the images are displayed by the displayed by the displaydevice 130.

<Conclusion>

The image evaluation apparatus 1100 pertaining to embodiment 2 performsevent evaluation of images included in an image group constituted of atleast two images according to the frequency of appearance, in the imagegroup, of images characterized by each different clothing type and thesimilarity between combinations of clothing in each image in the imagegroup.

Even when, for instance, clothing that a person appearing in a givenimage, which actually belongs to the “school gym uniform” clothing type,is not specified as belonging to the “school gym uniform” clothing typeas a result of incorrect recognition, there is a possibility that theimage evaluation apparatus 1100 pertaining to the present embodiment cancorrectly specify the clothing as belonging to the “school gym uniform”clothing type by referring to the similarity between imagecharacteristics of clothing appearing in the images. In short, the imageevaluation apparatus 1100 pertaining to the present embodiment iscapable of evaluating images with higher accuracy compared to the imageevaluation apparatus 100 pertaining to embodiment 1.

Embodiment 3

In embodiment 1, event evaluation of images is performed by estimatingthe clothing that people appearing in the images are wearing based ononly the information pertaining to the clothing appearing in the images,recognition of which is performed by the clothing recognition unit 106.In embodiment 3, event evaluation of images is performed by executing,in addition to the processing in embodiment 1, identification of a sameperson appearing in different images by performing clustering accordingto image characteristics of faces appearing in the images. Suchprocessing is executed in embodiment 3 based on the assumption that agiven person wears the same clothing throughout a given event. Note thatin the following, structures and data similar to those in embodiment 1are provided with the same reference signs and description thereon isomitted.

<Structure>

In the following, description is provided on an image evaluationapparatus 1500 pertaining to the present embodiment. FIG. 15 is afunctional block diagram of the image evaluation apparatus 1500 inembodiment 3. In the image evaluation apparatus 1500, the image eventevaluation unit 107, the storage unit 108, and the image informationacquisition unit 110 in the image evaluation apparatus 100 in embodiment1 are respectively replaced with an image event evaluation unit 1503, astorage unit 1504, and an image information acquisition unit 1510. Theimage information acquisition unit 1510 includes, in addition to thestructure of the image information acquisition unit 110, a facecharacteristics extraction unit 1501 and a face clustering unit 1502.

The face characteristics extraction unit 1501 extracts imagecharacteristics pertaining to a face (hereinafter referred to as faceimage characteristics) from each face region detected by the facedetection unit 103. The face characteristics extraction unit 1501manages the face image characteristics so extracted in association withthe corresponding face region.

The face clustering unit 1502 performs clustering according to the faceimage characteristics extracted by the face characteristics extractionunit 1501. Here, the clustering is performed such that faces appearingin a same image group that have similar face image characteristics areclassified into the same cluster. The face clustering unit 1502 provideseach of the clusters obtained as a result of the clustering with aunique cluster ID, and registers the cluster IDs to a table 1601illustrated in FIG. 16, which indicates faces and a clustercorresponding to each face. The table 1601 is stored by the storage unit1504. As indicated in the table 1601, faces classified into the samecluster can be estimated as corresponding to the same person.

The image event evaluation unit 1503 performs event evaluation of imagesin an image group according to the contents of the image managementinformation table 201, the contents of the people management informationtable 301, and the contents of the table 1601 indicating faces and acluster corresponding to each face. Details of how event evaluation isperformed are described later.

The storage unit 1504 stores the table 1601 indicating faces and acluster corresponding to each face, a table 1901 illustrated in FIG. 19indicating clusters and faces belonging to each cluster, and a table2001 illustrated in FIG. 20 indicating faces belonging to a cluster anda first clothing type for each face, in addition to the image managementinformation table 201, the people management information table 301, thetable 801 indicating third clothing types and the number of imagescharacterized by each third clothing type, the table 901 indicating acorrespondence between clothing types and events, and the table 1001indicating results of event evaluation performed with respect to imagegroups. Detailed description on the tables 1901 and 2001 is provided inthe following.

<Operations>

In the following, description is provided on operations of the imageevaluation apparatus 1500 pertaining to the present embodiment withreference to the flowchart illustrated in FIG. 17. In the following,description on the processing performed in each of Steps S601 throughS606 is omitted for being similar as the corresponding processing inembodiment 1.

The face characteristics extraction unit 1501 extracts face imagecharacteristics from each of the face regions detected by the facedetection unit 103 (Step S1701).

The face clustering unit 1502, according to the face imagecharacteristics extracted by the face characteristics extraction unit1501, classifies faces having similar face image characteristics intothe same cluster (Step S1702). The face clustering unit 1502 provides aunique cluster ID to each cluster obtained as a result of theclustering, and registers the cluster IDs to the table 1601 indicatingfaces and a cluster corresponding to each face.

The image event evaluation unit 1503 performs event evaluation of imagegroups based on the contents of the image management information table201, the contents of the people management information table 301, andthe contents of the table 1601 indicating faces and a clustercorresponding to each face (Step S1703). FIG. 18 is a detailed flowchartillustrating operations involved in Step S1703, where the image eventevaluation unit 1503 performs event evaluation of one evaluation-targetimage group.

The image event evaluation unit 1503 selects, from among the imagesincluded in the evaluation-target image group, images each including atleast a predetermined number of people (Step S701).

The image event evaluation unit 1503 selects one target image from amongthe images selected in Step S701 (Step S1801).

The image event evaluation unit 1503 selects one face appearing in theimage selected in Step S1801 (Step S1802).

The image event evaluation unit 1503 extracts, from each face belongingto a cluster to which the face selected in Step S1802 belongs, a firstclothing type specified by the clothing recognition unit 106 (StepS1803).

In the following, description is provided on a specific example of theoperations involved in Step S1803, with reference to the table 1601indicating faces and a cluster corresponding to each face. In thisexample, a face identified by face ID 1 is selected in Step S1802.According to the table 1601, the cluster into which the personidentified by face ID 1 has been classified is indicated by cluster ID1. The image event evaluation unit 1503 extracts faces corresponding tocluster ID 1 from the table 1601, and creates the table 1901 illustratedin FIG. 19, which indicates clusters and faces belonging to eachcluster. According to the table 1901, the people corresponding tocluster ID 1 are identified by face IDs 1, 13, 17, and 31. The imageevent evaluation unit 1503 extracts first clothing types specified fromthe faces identified by face IDs 1, 13, 17, and 31 from the peoplemanagement information table 301 illustrated in FIG. 3. FIG. 20illustrates an example of results of the extraction of first clothingtypes as described above.

The image event evaluation unit 1503 specifies a clothing typecharacterizing the cluster to which the face selected in Step S 1802belongs (hereinafter referred to as a “fourth clothing type”) byspecifying a clothing type satisfying a seventh criterion (Step S1804).In other words, the image event evaluation unit 1503, in Step S1804,specifies the clothing that a person corresponding to the face selectedin Step S1802 is wearing. Here, a first clothing type satisfying theseventh criterion is a first clothing type having been specified asbelonging to a specific clothing type by the clothing recognition unit106 among the first clothing types extracted through the processing inStep S1803. Here, when two or more first clothing types simultaneouslysatisfy the seventh criterion, the image event evaluation unit 1503specifies, as the fourth clothing type for the cluster to which the faceselected in Step S1802 belongs, a first clothing type common to arelatively great number of faces belonging to the cluster to which theface selected in Step S1802 belongs.

The image event evaluation unit 1503 determines whether or not theprocessing in Steps S1803 and S1804 has been performed with respect toall of the faces appearing in the image selected in Step S1801 (StepS1805). Processing proceeds to Step S1806 when determining affirmativelyin Step S1805 while processing returns to Step S1802 when determiningnegatively in Step S1805.

When determining affirmatively in Step S1805, the image event evaluationunit 1503 specifies a clothing type characterizing the image selected inStep S1801 (i.e., a third clothing type for the image selected in StepS1801) by specifying a clothing type in the image satisfying an eighthcriterion according to the fourth clothing types for the faces includedin the image (Step S1806). Here, a given fourth clothing type, among thefourth clothing types specified in Step S1804, is determined assatisfying the eighth criterion when, for instance, a ratio of thenumber of clothing of the given fourth clothing type to the total numberof clothing appearing in the image selected in Step S1801 exceeds 0.5.

Following this point, the image event evaluation unit 1503 performsprocessing corresponding to the processing performed by the image eventevaluation unit 107 in Steps S703 through S706 in embodiment 1.Therefore, description concerning the processing performed by the imageevent evaluation unit 1503 following this point is provided in asimplified manner.

The image event evaluation unit 1503 counts the number of images in theimage group characterized by the same third clothing type (Step S1807).

The image event evaluation unit 1503 determines whether or not theabove-described processing has been performed with respect to all of theimages having been selected in Step S701 (Step S1808). Processingproceeds to Step S1809 when determining affirmatively in Step S1808while processing returns to Step S1801 when determining negatively inStep S1808.

When determining affirmatively in Step S1808, the image event evaluationunit 1503 specifies a clothing type characterizing the image group(i.e., the second clothing type) by specifying a clothing typesatisfying a ninth criterion in the image group (Step S1809). Here, agiven third clothing type is determined as satisfying the ninthcriterion when, for instance, a ratio of the number of imagescharacterized by the given third clothing type to the total number ofimages of the image group selected in Step S701 exceeds 0.5.

The image event evaluation unit 1503 performs event evaluation withrespect to the image group according to the second clothing type (StepS1810).

The image evaluation apparatus 1500 performs event evaluation asdescribed above with respect to all of the image groups having beengenerated. Finally, the image evaluation apparatus 1500 outputs each ofthe images acquired by the image acquisition unit 101 to the displaydevice 130 such that the results of the evaluation performed withrespect to the images are displayed by the display device 130.

<Conclusion>

The image evaluation apparatus 1500 pertaining to embodiment 3 performsevent evaluation of images included in an image group constituted of atleast two images according to the clothing types appearing in the imagesin the image group, the frequency of appearance, in the image group, ofimages characterized by each different clothing type, and the results ofthe clustering performed with respect to the faces appearing in theimages.

Even when, for instance, a person who is wearing clothing that belongsto the “school gym uniform” clothing type is not recognized as wearingclothing belonging to the “school gym uniform” in a given image as aresult of incorrect recognition, the image evaluation apparatus 1500pertaining to the present embodiment is capable of correctly specifyingthe clothing that the person is wearing in the given image as belongingto the “school gym uniform” clothing type provided that the same personis recognized as wearing clothing belonging to the “school gym uniform”clothing type in another image in the image group. In short, the imageevaluation apparatus 1500 pertaining to the present embodiment iscapable of evaluating images with higher accuracy compared to the imageevaluation apparatus 100 pertaining to embodiment 1.

Embodiment 4

In embodiments 1 through 3, event evaluation of images in an image groupis performed according to the number of images in the image groupcharacterized by each different clothing type. In embodiment 4, eventevaluation of images is performed by utilizing the clustering of facesappearing in the images as described in embodiment 3, and according tothe number of people appearing in the images characterized by eachdifferent clothing type. Note that in the following, structures and datasimilar to those in embodiments 1 and 3 are provided with the samereference signs and description thereon is omitted.

<Structure>

In the following, description is provided on an image evaluationapparatus 2100 pertaining to the present embodiment. FIG. 21 is afunctional block diagram of the image evaluation apparatus 2100 inembodiment 4. In the image evaluation apparatus 2100, the image eventevaluation unit 1503 and the storage unit 1504 in the image evaluationapparatus 1500 in embodiment 3 are respectively replaced with an imageevent evaluation unit 2101 and a storage unit 2102.

The image event evaluation unit 2101 performs event evaluation of imagesin an image group according to the contents of the image managementinformation table 201, the contents of the people management informationtable 301, and the contents of the table 1601 indicating clusters andfaces belonging to each cluster. Details of how event evaluation isperformed are described later.

The storage unit 2012 stores a table 2401 illustrated in FIG. 24indicating third clothing types and the number of clusters characterizedby each third clothing type, in addition to the image managementinformation table 201, the people management information table 301, thetable 901 indicating a correspondence between clothing types and events,the table 1001 indicating results of event evaluation performed withrespect to image groups, the table 1601 indicating faces and a clustercorresponding to each face, the table 1901 indicating clusters and facesbelonging to each cluster, and the table 2001 indicating faces belongingto a cluster and a first clothing type for each face. Detaileddescription on the table 2401 is provided in the following.

<Operations>

In the following, description is provided on operations of the imageevaluation apparatus 2100 pertaining to the present embodiment, withreference to the flowchart illustrated in FIG. 22. In the following,description on the processing performed in each of Steps S601 throughS606 and Steps S1701 and S1702 is omitted for being similar as thecorresponding processing in embodiment 3.

The image event evaluation unit 2101 performs event evaluation of imagegroups based on the contents of the image management information table201, the contents of the people management information table 301, andthe contents of the table 1601 indicating faces and a clustercorresponding to each face (Step S2201). FIG. 23 is a detailed flowchartillustrating operations involved in Step S2201, where the image eventevaluation unit 2101 performs event evaluation of one evaluation-targetimage group.

The image event evaluation unit 2101 selects one cluster among one ormore clusters into which faces appearing in images in theevaluation-target image group are classified (Step S2301).

The image event evaluation unit 2101 extracts, from each face belongingto the cluster selected in Step S2301, a first clothing type specifiedby the clothing recognition unit 106 (Step S2302).

The image event evaluation unit 2101 specifies a clothing typecharacterizing the cluster selected in Step S2301 (i.e., the thirdclothing type) by specifying a clothing type satisfying a tenthcriterion (Step S2303). Here, a first clothing type satisfying the tenthcriterion is a first clothing type having been specified as belonging toa specific clothing type by the clothing recognition unit 106 among thefirst clothing types extracted through the processing in Step S2302.When two or more first clothing types simultaneously satisfy the tenthcriterion, the image event evaluation unit 2101 specifies, as the thirdclothing type for the cluster selected in Step S2301, a first clothingtype common to a relatively great number of faces belonging to thecluster selected in Step S2301, among the first clothing types extractedin Step S2302.

The image event evaluation unit 2101 counts the number of clusterscharacterized by each different third clothing type (Step S2304). Here,by counting the number of clusters characterized by each different thirdclothing type in the above described manner, the image event evaluationunit 2101 is actually counting, for each of the different clothingtypes, the number of people appearing in the image group that arewearing clothing belonging to the clothing type. More specifically, theimage event evaluation unit 2101 creates the table 2401 illustrated inFIG. 24 indicating third clothing types and the number of clusterscharacterized by each third clothing type. Further, when a given thirdclothing type is specified as characterizing a given cluster in StepS2303, the image event evaluation unit 2101 increments, by one, thevalue in the field indicating the number of clusters characterized bythe given clothing type.

Following this, the image event evaluation unit 2101 determines whetheror not the processing in Steps S2302 through S2304 has been performedwith respect to each of the clusters in the image group (Step S2305).Processing proceeds to Step S2306 when determining affirmatively in StepS2305 while processing returns to Step S2301 when determining negativelyin Step S2305.

When determining affirmatively in Step S2305, the image event evaluationunit 2101 specifies a clothing type characterizing the image group(i.e., the second clothing type) by specifying a clothing typesatisfying an eleventh criterion in the image group (Step S2306). Here,a given third clothing type is determined as satisfying the eleventhcriterion when, for instance, a ratio of the number of clusterscharacterized by the given third clothing type to the total number ofclusters in the image group exceeds 0.5. That is, when more than half ofthe people appearing in the image group are wearing clothing belongingto a given clothing type, the image event evaluation unit 2101 specifiesthe given clothing type as the second clothing type characterizing theimage group.

In the following, description is provided on a specific example of theprocessing in Step S2306, with reference to the table 2401 illustratedin FIG. 24, which indicates third clothing types and the number ofclusters characterized by each third clothing type. In this example, thetotal number of clusters in the image group is 10. Further, the ratio ofthe number of clusters characterized by the “school gym uniform”clothing type to the total number of clusters in the image group iscalculated as 6/10=0.6. Since the ratio (0.6) of the number of clusterscharacterized by the “school gym uniform” clothing type (6) to the totalnumber of clusters in the image group (10) exceeds a predetermined ratio(0.5), the “school uniform” clothing type satisfies the eleventhcriterion. As such, the image event evaluation unit 2101 specifies the“school gym uniform” clothing type as the second clothing type for theimage group.

The image event evaluation unit 2101 performs event evaluation withrespect to the image group according to the second clothing type (StepS2307).

The image evaluation apparatus 2100 performs event evaluation asdescribed above with respect to all of the image groups having beengenerated. Finally, the image evaluation apparatus 2100 outputs each ofthe images acquired by the image acquisition unit 101 to the displaydevice 130 such that the results of the evaluation performed withrespect to the images are displayed by the display device 130.

<Conclusion>

The image evaluation apparatus 2100 pertaining to embodiment 4 performsevent evaluation of images included in an image group constituted of atleast two images by classifying people appearing in different imagescorresponding to the same person into the same cluster, and according tothe number of clusters corresponding to people characterized by eachdifferent clothing type.

The image evaluation apparatus 1500 pertaining to embodiment 3 performsevent evaluation with respect to an image group in units of images. Assuch, when a given person appears in a plurality of images, the riskincreases of the evaluation results being influenced to a great extentby the given person as the number of images in which the given personappears increases. In contrast, the image evaluation apparatus 2100pertaining to the present embodiment classifies people appearing indifferent images corresponding to the same person into the same cluster,and performs evaluation in units of clusters. As such, the imageevaluation apparatus 2100 pertaining to the present embodiment iscapable of performing event evaluation of images included in an imagegroup while ensuring that the evaluation results are not stronglyinfluenced by a specific person.

<Supplement 1>

Up to this point, description has been provided on the present inventionwith reference to specific embodiments thereof. However, the presentinvention should not be construed as being limited to such embodiments.At least such modifications as presented in the following are to beconsidered as being within the spirit and scope of the presentinvention.

(1) In the embodiments, the image group generation unit 102 generatesimage groups by using a photography date/time acquired from the EXIFinformation provided to each of the acquired images. However, thepresent invention is not limited to this, and other methods may beapplied for the generation of image groups. For instance, the generationof image groups may be performed such that images taken within apredetermined distance from a given location are classified into thesame image group by using photography locations of images acquirablefrom metadata, such as the EXIF information.

(2) In the embodiments, the clothing detection unit 104 detects a firstperson whose face region is detected as being lower in a heightdirection of an image as appearing in front, in the camera direction, ofa second person whose face region is detected as being higher in theheight direction of the image. However, the present invention is notlimited to this, and other methods may be applied for the detection ofpositional relationships between people appearing in images. Forinstance, detection may be performed such that a first person whose faceregion occupies a relatively great area in an image compared to a secondperson is detected as appearing in front of the second person in thecamera direction. In addition, when the photography device 120 iscapable of taking and storing images such as stereograms includingparallax information, the positional relationships between peopleappearing in images may be detected by calculating distances betweenphotography-subjects and the camera.

(3) In the embodiments, the specification of clothing types by theclothing recognition unit 106 is performed by providing a classifierthat is capable of determining clothing types by being subjected tolearning according to the SVM method in advance. However, the presentinvention is not limited to this, and other methods may be applied asthe method for specifying clothing types. For instance, a clothing typemay be specified by performing matching between image characteristicshaving been extracted from a given clothing region and imagecharacteristics serving as templates of different clothing types.

In addition, the image evaluation apparatus pertaining to the presentinvention may be additionally provided with an update informationacquisition unit. An image evaluation apparatus provided with the updateinformation acquisition unit may acquire information for updating theclassifier, the templates, etc., via a network and may perform updatingof the classifier, the templates, etc., by using such information. Theprovision of such a structure realizes updating clothing types that theimage evaluation apparatus is capable of specifying as necessary.Further, the update information acquisition unit may further beconfigured such that, when updating the classifier, the templates, etc.,the update information acquisition unit also acquires information forupdating the table 901 indicating the correspondence between clothingtypes and events in accordance with the changes made to the classifier,the templates, etc. By making such a configuration, the events that theimage evaluation apparatus is able to associate with images can bechanged. That is, when provided with such a structure, the imageevaluation apparatus is capable of performing event evaluation such thatan image group is determined as being related to an event correspondingto a clothing type that has been newly added to the clothing types thatthe image evaluation apparatus is able to specify.

(4) In the embodiments, the similarity degree calculation unit 1101calculates a cosine similarity between two vectors each representingimage characteristics of clothing worn by a corresponding person as thesimilarity degree between a combination of clothing appearing in animage. However, the present invention is not limited to this, and othermethods may be applied in the calculation of a similarity degree betweena combination of clothing appearing in an image. For instance, thesimilarity degree calculation unit 1101 may calculate a Pearsonproduct-moment correlation coefficient between image characteristics ofclothing included in a combination of clothing as the similarity degreebetween the combination of clothing, or may calculate an reciprocal of asum of one and a Euclidean distance between two vectors eachrepresenting image characteristics of clothing included in thecombination of clothing as the similarity degree between the combinationof clothing.

(5) In Step S701 in embodiments 1 and 3, images in which only one personappears are excluded from consideration. However, the present inventionis not limited to this, and images in which only one person appears mayalso be selected as the target image in S701.

(6) With regards to the specification of a clothing type characterizinga given image by using similarity degrees calculated by the similaritydegree calculation unit 1101, in embodiment 2, the specification of aclothing type characterizing a given image is performed according to (i)the number of combinations of two people wearing clothing indicatingsimilarity or (ii) an average value of similarity degrees forcombinations of two people wearing clothing indicating similarity.However, the present invention is not limited to this, and other methodsmay be applied for specifying a clothing type characterizing a givenimage by using similarity degrees.

For instance, when there exists a combination of clothing having ahigher similarity degree than a predetermined threshold value and when,in the combination, clothing worn by a first person has been specifiedas belonging to a specific clothing type by the clothing recognitionunit 106 while the clothing worn by a second person has been specifiedas not belonging to a specific clothing type, the clothing worn by thesecond person can be treated as belonging to the same clothing type asthe clothing worn by the first person. In such a manner, compensation ofthe recognition results of the clothing recognition unit 106 may beperformed. When compensation of the recognition results is performed asdescribed above, the image evaluation apparatus pertaining to thepresent invention may specify a given clothing type whose number ofappearance in a given image satisfies a predetermined criterion as theclothing type characterizing the given image by using the compensatedrecognition results.

(7) In embodiments 1 through 3, the specification of a clothing typecharacterizing an image group is performed in accordance with the numberof images in the image group characterized by the same clothing type.However, the present invention is not limited to this, and other methodsmay be applied for specifying a clothing type characterizing an imagegroup according to the clothing types characterizing images in the imagegroups. For instance, a weight maybe provided to each of image in animage group according to the number of people appearing in each image,and evaluation of the images in the image group may be performed bysumming the weighted values for each of the images characterized by agiven clothing type. According to such a method, the images in which arelatively large number of people appear are provided with highevaluations, and therefore, specification can be made of a clothing typethat is most dominant among clothing types worn by a large number ofpeople appearing in the images.

(8) In embodiment 3, a clothing type appearing frequently in a givenimage is specified as the clothing type characterizing the given image.However, in a modification of the present invention, the specificationof a clothing type characterizing a given image may be performed bycalculating a degree of importance (hereinafter referred to as an“importance degree”) of each of the people appearing in the given imageand while taking into consideration the importance degrees socalculated. For instance, the calculation of importance degrees forpeople appearing in images may be performed such that, when a givenperson appears in a large number of the images, the given person isprovided with a high importance degree. In such a case, the calculationof importance degrees is performed in accordance with the number offaces classified into each cluster. Alternatively, the calculation ofimportance degrees of people may be performed such that, when a givenperson appears in the center of an image or appears occupying a greatarea in an image, the given person is provided with a high importancedegree. In such a case, the calculation of importance degrees isperformed in accordance with the positions of people in images or thesizes at which people are to be displayed.

In the following, description is provided on a modification of thepresent invention taking as an example the case where the specificationof a clothing type characterizing a given image is performed accordingto importance degrees of people appearing in the given image and wherethe importance degrees of people are determined in accordance with thenumber of faces classified into each cluster.

The face clustering unit 1502, after having performed clustering in StepS1702, manages the number of faces classified into each of the clustersobtained as a result of the clustering and the fourth clothing type foreach of the clusters (i.e., the clothing type characterizing each of theclusters). Note that, although the fourth clothing types are specifiedby the image event evaluation unit 1503 in embodiment 3, thespecification of the fourth clothing types may be alternativelyperformed by the face clustering unit 1502. In this example, the resultof the clustering indicates that cluster 1 includes six faces and ischaracterized by the “school gym uniform” clothing type, cluster 2includes two faces and is characterized by an “unspecified” clothingtype, and cluster 3 includes two faces and is characterized by the“unspecified” clothing type. Here, when a cluster is characterized bythe “unspecified” clothing type, the clothing type characterizing thecluster is a clothing type specified as not belonging to any specificclothing type.

The image event evaluation unit 1503, when three people classified intocluster 1, cluster 2, and cluster 3 appear in the image selected in StepS1801, specifies a clothing type characterizing the image in Step S1806according to the method described in the following.

First, the image event evaluation unit 1503 calculates an importancedegree (referred to hereinafter as a “cluster importance degree”) foreach person (cluster) appearing in the image. A cluster importancedegree for a given cluster is calculated, for instance, according to thenumber of faces belonging to the cluster. As such, in this example, thecluster importance degree for cluster 1 is six, the cluster importancedegree for cluster 2 is two, and the cluster importance degree forcluster 3 is two.

Subsequently, the image event evaluation unit 1503 calculates an imageclothing importance degree for clothing worn by each person appearing inthe image. The image clothing importance degree is, for instance, avalue calculated by normalizing an accumulated value of the clusterimportance degrees of the clusters characterized by a given clothingtype.

In this example, since cluster 1 is characterized by the “school gymuniform” clothing type and clusters 2 and 3 are characterized by the“unspecified” clothing type, the image clothing importance degree forthe “school gym uniform” clothing type is calculated as 6/(6+2+2)=0.6and the image clothing importance degree for the “unspecified” clothingtype is calculated as (2+2)/(6+2+2)=0.4.

Finally, the image event evaluation unit 1503 specifies a clothing typesatisfying a predetermined criterion (for instance, a clothing typewhose image clothing importance degree exceeds 0.5) as the clothing typecharacterizing the image. In this example, since the image clothingimportance degree of the “school gym uniform” clothing type is 0.6 andtherefore exceeds 0.5, the “school gym uniform” clothing type isspecified as the clothing type characterizing the given image.

In addition, in another modification, a clothing type characterizing agiven image may be specified by using only people provided with highimportance degrees (i.e., important people). Here, a personcorresponding to a cluster having a cluster importance degree of four orhigher is determined as an important person.

In this example, similar as in the above-described example, cluster 1includes six faces and is characterized by the clothing type “school gymuniform”, cluster 2 includes two faces and is characterized by the“unspecified” clothing type, and cluster 3 includes two faces and ischaracterized by the “unspecified” clothing type. As such, only theperson corresponding to cluster 1 is determined as an important personin the given image. In this example, a clothing type is specified ascharacterizing the given image when the clothing type satisfies apredetermined criterion. A given clothing type is specified assatisfying the predetermined criterion when a ratio of the number ofimportant people in the given image wearing clothing belonging to thegiven clothing type to the total number of important people appearing inthe image exceeds 0.5. In this example, the ratio of important peoplewearing clothing belonging to the “school gym uniform” clothing type tothe total number of important people appearing in the given image iscalculated as 1/1=1. Hence, the “school gym uniform” clothing typessatisfies the predetermined criterion, and therefore, the “school gymuniform” clothing type is specified as the clothing type characterizingthe given image.

The modifications described above are also applicable in thespecification of a clothing type characterizing an image group inembodiment 4. That is, the specification of a clothing typecharacterizing an image group in embodiment 4 may be performed whileproviding weights to clusters according to cluster importance degrees orwhile using only important people appearing in the image group.

When such modifications as described above are applied, people who areassumed as being important for the photographer of the evaluation-targetimages are provided with high importance degrees. As such, eventevaluation of images can be performed in accordance with the intentionsof the photographer of the evaluation-target images.

Note that importance degrees of people appearing in images may becalculated according to information pertaining to an individual who isto be provided with a high importance degree, which is acquirable fromexternal sources, such as a Social Networking Service (SNS). Forinstance, when face image data for a given person is acquired from anexternal source as information pertaining to the given person, who is tobe provided with a high importance degree, the given person may actuallybe provided with a high importance degree by extracting face imagecharacteristics from the face image data, and by performing matchingbetween the face image characteristics so obtained and the face imagecharacteristics of people classified into clusters.

(9) In each of embodiments 1 through 4, an image evaluation apparatuspertaining to each embodiment acquires a group of images from thephotography device 120 implemented as a digital camera or the like.However, the present invention is not limited to this, and the sourcefrom which images are acquired may be any source provided that thesource that has the function of accumulating images. For instance, theimage evaluation apparatus pertaining to the present invention may beconfigured to acquire a group of images recorded onto a recording mediumsuch as a hard disk.

(10) In each of embodiments 1 through 4, the image acquisition unit 101acquires a group of images accumulated by the photography device 120 ina collective manner. However the present invention is not limited tothis. For instance, the image acquisition unit 101 may acquire aspecific group of images, among the images accumulated in thephotography device 120, from the photography device 120 by making aspecification of certain conditions, such as a production date/time, andby acquiring a group of images satisfying such conditions.

(11) In each of embodiments 1 through 4, an image evaluation apparatuspertaining to each embodiment performs event evaluation with respect toeach of the image groups having been generated, and outputs each of theimages acquired from the image acquisition unit 101 to the displaydevice 130 such that the results of the evaluation performed withrespect to the images are displayed by the display device 130. However,the present invention is not limited to this, and the results of theevaluation performed by the image evaluation apparatus pertaining to thepresent invention may be utilized in other ways. For instance, in atable (a database) indicating events determined as corresponding toevaluation-target images and storage locations (addresses) of imagefiles corresponding to the evaluation-target images, the results of theevaluation may be used as indexes in a file system.

(12) In each of embodiments 1 through 4, an image evaluation apparatuscorresponding to each embodiment associates, in one-to-onecorrespondence, an image group with an event determined as being relatedto the image group as a result of event evaluation. However, the presentinvention is not limited to this, and an image group may be put intoassociation with a plurality of candidate events. For instance, in thetable 901 indicating the correspondence between clothing types andevents, a plurality of candidate events may be associated with eachclothing type. Further, based on such a table, an image group specifiedas being characterized by a given second clothing type may be associatedwith multiple candidate events that correspond to the given secondclothing type.

In addition, a plurality of second clothing types may be specified ascharacterizing a given image group according to the number of images inthe image group characterized by each third clothing type, and the givenimage group may be associated with a plurality of events associated withthe plurality of second clothing types. In such a case, the results ofevent evaluation performed with respect to the image group may bedisplayed in the form of a ranking in which the events are ranked inaccordance with the number of images in the image group characterized byeach third clothing type.

Further, when multiple second clothing types are to be specified ascharacterizing a given image group, event evaluation of the image groupmay be performed in accordance with a table separately prepared in whicha combination of second clothing types such as a “suit” clothing typeand a “dress” clothing type and an event such as a “party” are put intoassociation. In such a case, event evaluation of the given image groupis performed according to the combination of second clothing typesspecified for the given image group.

(13) Specific ones of the above-described embodiments and modificationsmay be used in combined.

(14) A control program composed of program code in machine language orhigh-level language for causing a processor of an image evaluationapparatus and various circuits connected to the processor to execute theprocessing described in each of the embodiments 1 through 4 may bedistributed by recording the control program onto recording media, or bytransmitting the control program via various communication paths. Suchrecording media which may be used in the distribution of the controlprogram include IC cards, hard disks, optical discs, flexible disks,ROMs, flash memories, and the like. The distributed control program isto be stored to a memory or the like which may be read by a processor,so that the processor may execute the control program. Thereby, each ofthe functions described in the embodiments is to be realized. Note thatthe processor may execute the control program either directly, aftercompiling the control program, or with use of an interpreter.

(15) Each of the functional structures pertaining to embodiments 1through 4 may be implemented as an LSI (Large Scale Integration), whichis a type of integrated circuit. Each of such structures may beseparately integrated into a single chip, or the structures may beintegrated into a single chip so as to include a part or all of thestructures. Although description has been made on the basis of an LSI inthe above, the name of the integrated circuit may differ according tothe degree of integration of circuits. Other integrated circuits includean IC (integrated circuit), a system LSI, a super LSI, and an ultra LSI.Further, the method applied for forming integrated circuits is notlimited to the LSI, and integrated circuits may be formed by using adedicated circuit or a general purpose processor. In addition, thepresent invention may be realized by using an FPGA (Field ProgrammableGate Array) being an LSI which can be programmed after manufacturing, ora reconfigurable processor being a LSI, reconfiguration of which couldbe made to the connection of internal circuit cells and settings. Inaddition, the computation performed by such functional blocks may alsobe performed by using, for instance, a Digital Signal Processor (DSP) ora Central Processing Unit (CPU). Further, such processing steps asdescribed above may be executed by being recorded onto recording mediaas a control program and by such a control program being executed.

<Supplement 2>

In the following, description is provided on various aspects of thepresent invention and effects that are achievable by such aspects.

One aspect of the present invention is an image evaluation apparatusthat evaluates images in an image group, comprising: a firstspecification unit that specifies, for each person appearing in each ofthe images, a type of clothing that the person is wearing as a firstclothing type; a second specification unit that specifies a type ofclothing characterizing the image group as a second clothing typeaccording to a frequency of appearance of each of different firstclothing types specified from the images in the image group; and anevaluation unit that evaluates the images in the image group bydetermining, according to the second clothing type, an event at whichthe images in the image group have been taken.

(B) One aspect of the present invention is an image evaluation methodfor evaluating images in an image group, comprising: a firstspecification step of specifying, for each person appearing in each ofthe images, a type of clothing that the person is wearing as a firstclothing type; a second specification step of specifying a type ofclothing characterizing the image group as a second clothing typeaccording to a frequency of appearance of each of different firstclothing types specified from the images in the image group; and anevaluation step of evaluating the images in the image group bydetermining, according to the second clothing type, an event at whichthe images in the image group have been taken.

(C) One aspect of the present invention is a program for causing acomputer to execute image evaluation processing of evaluating images inan image group, the image evaluation processing comprising: a firstspecification step of specifying, for each person appearing in each ofthe images, a type of clothing that the person is wearing as a firstclothing type; a second specification step of specifying a type ofclothing characterizing the image group as a second clothing typeaccording to a frequency of appearance of each of different firstclothing types specified from the images in the image group; and anevaluation step of evaluating the images in the image group bydetermining, according to the second clothing type, an event at whichthe images in the image group have been taken.

(D) One aspect of the present invention is an integrated circuit thatevaluates images in an image group, comprising: a first specificationunit that specifies, for each person appearing in each of the images, atype of clothing that the person is wearing as a first clothing type; asecond specification unit that specifies a type of clothingcharacterizing the image group as a second clothing type according to afrequency of appearance of each of different first clothing typesspecified from the images in the image group; and an evaluation unitthat evaluates the images in the image group by determining, accordingto the second clothing type, an event at which the images in the imagegroup have been taken.

According to the structures in (A) through (D) above, even whenrecognition of the clothing that a person appearing in a given image iswearing is performed incorrectly, evaluation of the given image isperformed correctly due to reference being made to the frequency atwhich different types of clothing appear in a plurality of imagesincluding the given image.

(E) The image evaluation apparatus may further comprise a thirdspecification unit that specifies, for each image in the image group, atype of clothing characterizing the image as a third clothing typeaccording to one or more first clothing types corresponding to one ormore people appearing in the image, wherein the second specificationunit may specify the second clothing type for the image group accordingto the number of images in the image group characterized by each ofdifferent third clothing types.

According to the structure in (E) above, a clothing type characterizinga plurality of images is specified according to the number of images,among the plurality of images, characterized by each different clothingtype. As such, a clothing type appearing in a larger number of images,among the plurality of images, is specified.

(F) In the image evaluation apparatus, the third specification unit mayspecify, as a third clothing type for each image in the image group inwhich a plurality of people appear, one first clothing type, among aplurality of first clothing types corresponding to the plurality ofpeople appearing in the image, that corresponds to at least apredetermined proportion of the plurality of people appearing in theimage or corresponds to at least a predetermined number of people amongthe plurality of people appearing in the image.

(G) The image evaluation apparatus may further comprise a calculationunit that extracts, for each person appearing in each of the images,image characteristics of clothing that the person is wearing, andcalculates, for each possible combination of two people appearing ineach image in the image group in which a plurality of people appear, asimilarity degree indicating similarity between clothing that the twopeople are wearing according to image characteristics of the clothingthat the two people are wearing, wherein the third specification unitmay specify a third clothing type for each image in the image group inwhich a plurality of people appear according to a plurality of firstclothing types corresponding to the plurality of people appearing in theimage and a similarity degree for each possible combination of twopeople among the plurality of people appearing in the image.

(H) In the image evaluation apparatus, the third specification unit, foreach image in the image group in which a plurality of people appear, mayselect, from among one or more possible combinations of two people amongthe plurality of people appearing in the image, one or more combinationsof two people determinable as wearing similar clothing according to asimilarity degree for each of the one or more possible combinations, andwhen the one or more combinations so selected occupy at least apredetermined proportion of the one or more possible combinations or thenumber of the one or more combinations so selected equals at least apredetermined number, may specify a third clothing type for the imageaccording to a first clothing type for each person included in the oneor more combinations so selected.

(I) In the image evaluation apparatus, the third specification unit, foreach image in the image group in which a plurality of people appear, mayselect, from among one or more possible combinations of two people amongthe plurality of people appearing in the image, one or more combinationsof two people determinable as wearing similar clothing according to asimilarity degree for each of the one or more possible combinations, andwhen an average of one or more similarity degrees for the one or morecombinations so selected exceeds a value calculated by using apredetermined mathematical formula, may specify a third clothing typefor the image according to a first clothing type for each personincluded in the one or more combinations so selected.

According to the structures in (G) through (I) above, a clothing typecharacterizing a given image is specified by using, in addition toresults of recognition performed of clothing appearing in the givenimage, similarity degrees between the clothing appearing in the givenimage. As such, even in a case where, according to conventionaltechnology, clothing actually belonging to the same clothing type isspecified as belonging to different clothing types as a result ofincorrect recognition, there is a possibility that such clothing isspecified as belonging to the same clothing type as a result of therecognition of the clothing being performed by referring to thesimilarity degree between the clothing.

(J) The image evaluation apparatus may further comprise: aclassification unit that extracts, for each person appearing in each ofthe images, image characteristics of a face of the person, andclassifies a plurality of people appearing in the images into aplurality of clusters according to similarity degrees indicatingsimilarity between face image characteristics of the plurality of peoplesuch that people appearing in different images but corresponding to thesame person are classified into the same cluster; and a fourthspecification unit that specifies, for each of the plurality ofclusters, a type of clothing characterizing the cluster as a fourthclothing type according to a first clothing type for each personclassified into the cluster, wherein the third specification unit mayspecify a third clothing type for each image in the image group in whicha plurality of people appear according to a fourth clothing type foreach cluster into which one or more of the plurality of people appearingin the image are classified.

According to the structure in (J) above, clustering is performedaccording to face image characteristics, and as a result, whereby peopleappearing in a plurality of images, who actually correspond to the sameperson, are identified as corresponding to the same person. As such,even when a type of clothing that the person is wearing in a given imageis incorrectly recognized, there is a possibility of the result of therecognition performed with respect to the given image being correctedaccording to the result of the recognition performed of the clothingthat the same person is wearing in a different image.

(K) In the image evaluation apparatus, the classification unit maycalculate an importance degree of each of the plurality of clusters, andthe third specification unit may specify a third clothing type for eachimage in the image group in which a plurality of people appear accordingto a fourth clothing type for each cluster into which one or more of theplurality of people appearing in the image are classified and animportance degree for each cluster into which one or more of theplurality of people appearing in the image are classified.

According to the structure in (K) above, the specification of a clothingtype characterizing a given image can be performed while taking intoconsideration the clothing that an important person is wearing in thegiven image.

(L) The image evaluation apparatus may further comprise: aclassification unit that extracts, for each person appearing in each ofthe images, image characteristics of a face of the person, andclassifies a plurality of people appearing in the images into aplurality of clusters according to similarity degrees indicatingsimilarity between face image characteristics of the plurality of peoplesuch that people appearing in different images but corresponding to thesame person are classified into the same cluster; and a thirdspecification unit that specifies, for each of the plurality ofclusters, a type of clothing characterizing the cluster as a thirdclothing type according to a first clothing type for each personclassified into the cluster, wherein the second specification unit mayspecify the second clothing type for the image group according to thenumber of clusters, among the plurality of clusters, characterized byeach of different third clothing types.

According to the structure in (L) above, a clothing type characterizinga plurality of images is specified according to the number of clusterscharacterized by each different clothing type. As such, specification isperformed of a type of clothing worn by many of the people appearing inthe images. Further, according to the structure in (L) above, even whena given person appears in many images, the risk is reduced of theevaluation results being influenced to a great extent by the givenperson.

(M) In the image evaluation apparatus, the classification unit maycalculate an importance degree of each of the plurality of clusters, andthe second specification unit may specify the second clothing typeaccording to the number of clusters, among the plurality of clusters,characterized by each of different third clothing types and theimportance degree for each of the plurality of clusters.

According to the structure in (M) above, an event related an image groupis determined while taking into consideration the clothing worn byimportant people.

(N) In the image evaluation apparatus, the first specification unit mayspecify a first clothing type for each person appearing in each of theimages by using clothing information usable for specifying clothingtypes, and the image evaluation apparatus may further comprise an updateunit that updates the clothing information.

According to the structure in (N) above, the clothing types that theimage evaluation apparatus is able to specify and the events that theimage evaluation apparatus is able to determine as being related toimage groups is changed as necessary.

(O) The image evaluation apparatus may further comprise a calculationunit that detects, for each person appearing in each of the images, aface region corresponding to a face of the person, and calculates aclothing region corresponding to clothing that the person is wearing,wherein the first specification unit may specify a first clothing typefor each person appearing in each of the images according to imagecharacteristics extracted from a clothing region corresponding to theperson.

According to the structure in (O) above, a clothing region for a personappearing in a given image is detectable when a face region, in thegiven image, for the person is detectable.

(P) In the image evaluation apparatus, in a given image among theimages, when two clothing regions corresponding to two people appearingin the given image overlap each other, the calculation unit may specify,according to two face regions in the given image corresponding to thetwo people, one person among the two people that appears in front of theother person and may determine a region of the given image at which thetwo clothing regions overlap as belonging to a clothing region of theone person so specified.

According to the structure in (P) above, when a region in a given imagedetected as the clothing a first person is wearing and a region in thesame image detected as the clothing a second person is wearing overlapeach other, the clothing region for each of the first person and thesecond person can be determined based on a positional relationshipbetween the two people in the given image.

INDUSTRIAL APPLICABILITY

The image evaluation apparatus pertaining to the present invention isapplicable to devices accumulating still images and moving images,photography devices such as digital cameras, mobile phones provided withcamera functions, and movie cameras, Personal Computers (PCs), and thelike.

REFERENCE SIGNS LIST

100, 1100, 1500, 2100 image evaluation apparatus

101 image acquisition unit

102 image group generation unit

103 face detection unit

104 clothing detection unit

105 clothing characteristics extraction unit

106 clothing recognition unit

107, 1102, 1503, 2101 image event evaluation unit

108, 1103, 1504, 2102 storage unit

110, 1110, 1510 image information acquisition unit

120 photography device

130 display device

201 image management information table

301 people management information table

401, 402, 403 image

402 a, 403 a, 403 b detected face region

402 a′, 403 a′, 403 b′ detected clothing region

801 table indicating third clothing types and the number of imagescharacterized by each third clothing type

901 table indicating correspondence between clothing types and events

1001 table indicating results of event evaluation performed with respectto image groups

1101 similarity degree calculation unit

1201 table indicating similarity between the clothing people in an imageare wearing

1501 face characteristics extraction unit

1502 face clustering unit

1601 table indicating faces and a cluster corresponding to each face

1901 table indicating clusters and faces belonging to each cluster

2001 table indicating faces belonging to a cluster and a first clothingtype for each face

2401 table indicating third clothing types and the number of clusterscharacterized by each third clothing type

1-16. (canceled)
 17. An image evaluation apparatus that evaluates images in an image group, comprising: a first specification unit that specifies, for each person appearing in each of the images, a type of clothing that the person is wearing as a first clothing type; a second specification unit that specifies, for each image in the image group, a type of clothing characterizing the image as a second clothing type according to one or more first clothing types corresponding to one or more people appearing in the image; a third specification unit that specifies a type of clothing characterizing the image group as a third clothing type according to the number of images in the image group characterized by each of different second clothing types; and an evaluation unit that evaluates the images in the image group by determining, according to the third clothing type, an event at which the images in the image group have been taken.
 18. The image evaluation apparatus of claim 17, wherein the second specification unit specifies, as a second clothing type for each image in the image group in which a plurality of people appear, one first clothing type, among a plurality of first clothing types corresponding to the plurality of people appearing in the image, that corresponds to at least a predetermined proportion of the plurality of people appearing in the image or corresponds to at least a predetermined number of people among the plurality of people appearing in the image.
 19. The image evaluation apparatus of claim 17 further comprising a calculation unit that extracts, for each person appearing in each of the images, image characteristics of clothing that the person is wearing, and calculates, for each possible combination of two people appearing in each image in the image group in which a plurality of people appear, a similarity degree indicating similarity between clothing that the two people are wearing according to image characteristics of the clothing that the two people are wearing, wherein the second specification unit specifies a second clothing type for each image in the image group in which a plurality of people appear according to a plurality of first clothing types corresponding to the plurality of people appearing in the image and a similarity degree for each possible combination of two people among the plurality of people appearing in the image.
 20. The image evaluation apparatus of claim 19, wherein the second specification unit, for each image in the image group in which a plurality of people appear, selects, from among one or more possible combinations of two people among the plurality of people appearing in the image, one or more combinations of two people determinable as wearing similar clothing according to a similarity degree for each of the one or more possible combinations, and when the one or more combinations so selected occupy at least a predetermined proportion of the one or more possible combinations or the number of the one or more combinations so selected equals at least a predetermined number, specifies a second clothing type for the image according to a first clothing type for each person included in the one or more combinations so selected.
 21. The image evaluation apparatus of claim 19, wherein the second specification unit, for each image in the image group in which a plurality of people appear, selects, from among one or more possible combinations of two people among the plurality of people appearing in the image, one or more combinations of two people determinable as wearing similar clothing according to a similarity degree for each of the one or more possible combinations, and when an average of one or more similarity degrees for the one or more combinations so selected exceeds a value calculated by using a predetermined mathematical formula, specifies a second clothing type for the image according to a first clothing type for each person included in the one or more combinations so selected.
 22. The image evaluation apparatus of claim 17 further comprising: a classification unit that extracts, for each person appearing in each of the images, image characteristics of a face of the person, and classifies a plurality of people appearing in the images into a plurality of clusters according to similarity degrees indicating similarity between face image characteristics of the plurality of people such that people appearing in different images but corresponding to the same person are classified into the same cluster; and a fourth specification unit that specifies, for each of the plurality of clusters, a type of clothing characterizing the cluster as a fourth clothing type according to a first clothing type for each person classified into the cluster, wherein the second specification unit specifies a second clothing type for each image in the image group in which a plurality of people appear according to a fourth clothing type for each cluster into which one or more of the plurality of people appearing in the image are classified.
 23. The image evaluation apparatus of claim 22, wherein the classification unit calculates an importance degree of each of the plurality of clusters, and the second specification unit specifies a second clothing type for each image in the image group in which a plurality of people appear according to a fourth clothing type for each cluster into which one or more of the plurality of people appearing in the image are classified and an importance degree for each cluster into which one or more of the plurality of people appearing in the image are classified.
 24. An image evaluation apparatus for evaluating images included in an image group, comprising: a first specification unit that specifies, for each person appearing in each of the images, a type of clothing that the person is wearing as a first clothing type; a classification unit that extracts, for each person appearing in each of the images, image characteristics of a face of the person, and classifies a plurality of people appearing in the images into a plurality of clusters according to similarity degrees indicating similarity between face image characteristics of the plurality of people such that people appearing in different images but corresponding to the same person are classified into the same cluster; a second specification unit that specifies, for each of the plurality of clusters, a type of clothing characterizing the cluster as a second clothing type according to a first clothing type for each person classified into the cluster; a third specification unit that specifies a type of clothing characterizing the image group as a third clothing type according to the number of clusters, among the plurality of clusters, characterized by each of different second clothing types; and an evaluation unit that evaluates the images in the image group by determining, according to the third clothing type, an event at which the images in the image group have been taken.
 25. The image evaluation apparatus of claim 24, wherein the classification unit calculates an importance degree of each of the plurality of clusters, and the third specification unit specifies the third clothing type according to the number of clusters, among the plurality of clusters, characterized by each of different second clothing types and the importance degree for each of the plurality of clusters.
 26. The image evaluation apparatus of claim 17, wherein the first specification unit specifies a first clothing type for each person appearing in each of the images by using clothing information usable for specifying clothing types, and the image evaluation apparatus further comprises an update unit that updates the clothing information.
 27. The image evaluation apparatus of claim 17 further comprising a calculation unit that detects, for each person appearing in each of the images, a face region corresponding to a face of the person, and calculates a clothing region corresponding to clothing that the person is wearing, wherein the first specification unit specifies a first clothing type for each person appearing in each of the images according to image characteristics extracted from a clothing region corresponding to the person.
 28. The image evaluation apparatus of claim 27, wherein in a given image among the images, when two clothing regions corresponding to two people appearing in the given image overlap each other, the calculation unit specifies, according to two face regions in the given image corresponding to the two people, one person among the two people that appears in front of the other person and determines a region of the given image at which the two clothing regions overlap as belonging to a clothing region of the one person so specified.
 29. An image evaluation method for evaluating images in an image group, comprising: a first specification step of specifying, for each person appearing in each of the images, a type of clothing that the person is wearing as a first clothing type; a second specification step of specifying, for each image in the image group, a type of clothing characterizing the image as a second clothing type according to one or more first clothing types corresponding to one or more people appearing in the image; a third specification step of specifying a type of clothing characterizing the image group as a third clothing type according to the number of images in the image group characterized by each of different second clothing types; and an evaluation step of evaluating the images in the image group by determining, according to the third clothing type, an event at which the images in the image group have been taken.
 30. A program for causing a computer to execute image evaluation processing of evaluating images in an image group, the image evaluation processing comprising: a first specification step of specifying, for each person appearing in each of the images, a type of clothing that the person is wearing as a first clothing type; a second specification step of specifying, for each image in the image group, a type of clothing characterizing the image as a second clothing type according to one or more first clothing types corresponding to one or more people appearing in the image; a third specification step of specifying a type of clothing characterizing the image group as a third clothing type according to the number of images in the image group characterized by each of different second clothing types; and an evaluation step of evaluating the images in the image group by determining, according to the third clothing type, an event at which the images in the image group have been taken.
 31. An integrated circuit that evaluates images in an image group, comprising: a first specification unit that specifies, for each person appearing in each of the images, a type of clothing that the person is wearing as a first clothing type; a second specification unit that specifies, for each image in the image group, a type of clothing characterizing the image as a second clothing type according to one or more first clothing types corresponding to one or more people appearing in the image; a third specification unit that specifies a type of clothing characterizing the image group as a third clothing type according to the number of images in the image group characterized by each of different second clothing types; and an evaluation unit that evaluates the images in the image group by determining, according to the third clothing type, an event at which the images in the image group have been taken. 